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The Auburn University Educational Leadership Team and its K-12 practitioner and 
community partners collaboratively developed a comprehensive, on-going evaluation plan 
to assess a newly implemented field-based graduate program. The purpose of the present 
manuscript was to describe the evaluation guidelines and plans. The leadership team and 
partners designed the assessment and evaluation plan to incoiporate and integrate 
components from a variety of evaluation approaches to address various program, 
stakeholder, and governing body data needs. By drawing on a variety of evaluation 
approaches the team and partners generated an array of data or sources of knowledge to 
better inform decisions about program implementation processes and outcome 
effectiveness and, thereby, enhance the team's ability to reform and reshape the program 
to meet on-going and changing stakeholder and participant needs. 


Program evaluation can be a 
daunting task considering the wide 
array of alternative models and 
approaches, ranging from early Tylerian 
objectives-oriented to recently 
developed participant-oriented 

approaches (Fitzpatrick, Sanders, & 
Worthen, 2004). Popham (1993) writes 
that "[t]o evaluate something is to 
appraise its quality" (p. 7). In order to 
develop an evaluation plan that 
provides the basis for program 
appraisal, answers to a number of 
questions are needed, including: What 
should be evaluated? Who should be 
involved in evaluation planning and 
decision making? What data should be 
collected? These questions address just a 


few issues facing educational program 
evaluators. 

The purpose of this manuscript is 
to describe an evaluation plan designed 
to address the data and decision-making 
needs of Auburn University's newly 
redesigned Educational Leadership 
Master's Degree Program that brings 
together as partners, or collaborators, 
Auburn University faculty, K-12 
administrators and practitioners, and 
Alabama State Department of Education 
(ALSDE) as both program creators and 
evaluators. The following sections 
include (a) an outline of assumptions, 
(b) an overview of program evaluation 
approaches, (c) a discussion of the 
literature on educational leadership 
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programs, and (d) the a description of 
the evaluation plan for the redesigned 
Auburn University Educational 
Leadership Program. 

Assumptions 

Wiggins (1992) writes that "we 
must be more critical in our diagnoses 
of problems and how problems change 
as we vary the paradigms used in the 
diagnosis. This necessitates 

systematically confronting and 
challenging existing paradigms, shifting 
paradigms, and reassessing paradigms 
lost (p. 9)." Based on the premise that 
paradigms 

shift, this author makes three major 
assumptions: 

(1) Education is a constantly 
changing and multi-faceted 
endeavor. This assumption 
implies that educators must be 
willing to constantly confront 
and challenge the status quo. 

(2) Because education is a complex 
and changing endeavor, the 
preparation of educational 
leaders is multi-faceted and 
complex and requires the 
willingness of faculty, students, 
and administrators to confront 
and challenge existing 
paradigms and methods and to 
change and rearrange the 
various components of a 
program, forming different 
patterns as does a kaleidoscope 
each time it is turned. 

(3) Consistently and continuously 
confronting and challenging 
multiple existing educational 
perspectives necessitates the 


use of multiple approaches to 
evaluation in order to 
accommodate different data 
uses and evaluation purposes. 

In the next section, a brief 
synopsis of various program evaluation 
approaches is provided and a discussion 
of underlying dimensions and 
implications in relation to the choice of 
evaluation tools and processes is given. 

Program Evaluation Approaches 

Fitzpatrick et al. (2004) indicate 
that during the era encompassing 1960 
through 1990, almost sixty approaches 
or evaluation proposals were 
developed. Differences in approaches 
imply differences in methodology and 
different evaluation needs lead to 
various evaluation approaches. 
Fitzpatrick et al. identify five 
classifications of evaluation approaches 
as described in the following 
paragraphs. 

Objectives-oriented Approaches. 
The major defining feature of objectives- 
oriented program evaluation 
approaches is that the program goals 
and objectives are explicitly stated and 
the evaluation activities assess the 
extent to which these goals and 
objectives have been met. Tyler (1991) is 
credited with conceptualizing this 
approach and others have expanded 
and refined it (Fitzpatrick et al., 2004; 
Popham, 1993; Stufflebeam & 
Shinkfield, 2007; Tyler, 1991). 

Management-oriented Approach. 
The management-oriented approach 
emphasizes the program decision¬ 
maker's need for evaluation results, 
reasoning that "evaluative information 
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is an essential part of good decision¬ 
making" (p. 88). A well-known 

management-oriented approach is the 
CIPP model proposed by Stufflebeam 
(Fitzpatrick et al., 2004; Stufflebeam & 
Shinkfield, 2007). The model name 
(CIPP) is an acronym based on a 
framework that addresses decisions 
related to Context (population, 
problems), Input (strategies, budgets, 
schedules), Process (procedures, 

refinement decisions), and Product 
(outcomes, program continuation 

decisions) (Stufflebeam & Shinkfield). 

Consumer-oriented Approach. 
The consumer-oriented approach seeks 
to inform consumers about products so 
that they (the consumers) have the 
informational basis for making 

judgments about human service 
products, such as commercial 
educational programs and materials. 
Consumer-oriented evaluators use a 
variety of data collection methods, 
including checklists, needs assessments, 
cost analysis, experimental and quasi- 
experimental designs, and codes 

(Stufflebeam & Shinkfield, 2007). 

Expertise-oriented Approaches. 
The expertise-oriented evaluator relies 
on his or her expert judgment about the 
worth of the object of evaluation (Eisner, 
1991; Fitzpatrick et al., 2004). For 
example, a leadership graduate program 
evaluation might rely on the expertise of 
faculty, school administrators, and 
teachers to judge the program's 
effectiveness. 

Participant-oriented Approaches. 
Participant-oriented approaches place 
much emphasis on the needs of the 
program participants. Robert Stake, an 
early proponent of this approach, began 


writing about a "responsive evaluation" 
approach in the 1970s (Fitzpatrick et al., 
2004). His major focus is attending and 
responding to the participant and 
stakeholder need for information. Out of 
Stake's responsive evaluation approach 
grew the participant-oriented 
approaches, including the utilization- 
focused evaluation that specifically 
attends to and gives power to the 
evaluation user (Patton, 1994) and 
empowerment evaluation (Fetterman, 

2002) , an approach that involves the 
participants in the evaluation process. 
This approach seems particularly useful 
in education as all members of a 
democratic society have a stake in the 
making sure the citizens are well 
educated in order to make informed 
societal decisions. 

The different approaches imply 
different underlying assumptions and 
views on how research should be 
conducted. House (1978) classifies the 
objectives, management, and consumer- 
oriented approaches as 'objectivist' 
(positivist/ post-positivist) and the 
expert and the expertise and participant- 
oriented approaches as 'subjectivist' or 
constructivist. Positivists and post¬ 
positivists rely more on quantitative 
methods whereas constructivists tend to 
use qualitative methods (Creswell, 

2003) . Fitzpatrick et al. (2004) purport 

that their classification schema is based 
on the "driving force behind doing the 
evaluation" (p. 68) including (a) 

questions addressed and (b) underlying 
organizers related to who is involved in 
the evaluation processes and 
decisions... evaluator, managers, 

stakeholders. The objectives-oriented 
and management-oriented approaches. 
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for example, emphasize the decision¬ 
making needs of those in charge while 
the other approaches, to varying 
degrees, consider or involve 
stakeholders. The Educational 
Leadership team, like Fitzpatrick et al. 
base their evaluation plan on questions 
and underlying organizers and integrate 
approaches. 

Integrating Approaches. Kirkhart 
(2000) proposes an integrated theory of 
evaluation influence that he conceives to 
be multifaceted. He conceptualizes three 
dimensions: intended versus 

unintended, process and results, and 
time (immediate, end-of cycle, and long¬ 
term). In planning the evaluation, the 
Auburn University Leadership Team 
decided upon a multifaceted approach 
that encompasses evaluation influences, 
but also evaluation needs, such as data 
needs of accreditation boards, hiring 
needs of public schools, and contexts 
such as the campus-based classroom 
and the field-based public school 
setting. The goal was to identify aspects 
or components of approaches that best 
serve the evaluation purposes and 
needs of the Educational Leadership 
Master's Degree Program. The 
integrated approach described in this 
paper is much like Patton's Utilization- 
focused evaluation (1997) in which 
various approaches are combined to 
form a distinct approach. For example, 
an accreditation system such as the 
National Council for Accreditation of 
Teacher Education (NCATE), an 
expertise-oriented approach to 
regulating educational institutions, 
holds the institution to standards 
developed by the field. Perhaps best 
classified as expertise-oriented, the 


accreditation system can also be 
conceptualized in terms of goals with 
standards representing the goals that 
are assessed. The participant-oriented 
approach involves stakeholders, such as 
students, in the evaluation process, 
which also supports the educational 
institution in understanding and 
attending to the needs of its consumers, 
the students. By drawing from and 
integrating different approaches into the 
evaluation plan, a wider variety of 
stakeholder and program needs are 
addressed in the evaluation plan. 

The first task of those developing 
an evaluation plan is to define criteria 
used for judging worth of either 
processes or products (Guskey, 1999). 
What do successful programs look like 
and what information is needed to make 
judgments about success? Thus, the 
initial step was to identify and review 
literature on leadership preparation 
programs. It became apparent that one 
difficulty in both designing and 
planning for evaluation, however, lies in 
the complexity of principal preparation 
programs as well as the varying, 
intertwined, and, at times, conflicting 
interests of program faculty, students, 
the K-12 institutions, and accreditation 
agencies. To further complicate 
planning for evaluation, even principal 
preparation programs considered to be 
'good' vary widely in content, design, 
and mode of delivery (Jackson & Kelley, 
2002; Wallace Foundation, 2008). In the 
next section the literature on principal 
preparation programs recommended 
program elements are discussed. 
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Designing and Evaluating Educational 
Leadership Programs 

A number of educational 
leadership experts and researchers 
(Barnett & Muth, 2008; Cambron- 
McCabe & McCarthy, 2005; Darling- 
Hammond, LaPointe, Meyerson, & Orr, 
2007; Hart & Pounder, 1999; Jackson & 
Kelley, 2002; Murphy, 2002; Orr, 2006; 
Wallace Foundation, 2008; Young, 
Petersen, & Short, 2002) studied 
innovative leadership preparation 
programs and, though programs looked 
quite different, specific program 
features considered to facilitate success 
were teased out. For example, Jackson 
and Kelley (2002) studied six principal 
preparation programs across the United 
States, and The Wallace Foundation 
(2008) commissioned a team of 
researchers to study 'exemplary' 
principal preparation programs based 
on "evidence of strong outcomes in 
preparing school leaders" (p. 2). 

Principal preparation program features 
that seemed to permeate 'innovative' or 
'exemplary' principal preparation 
programs included (a) alignment 
between curriculum and professional 
standards, (b) rigorous selection of 
candidates, (c) cohort structure, (d) 
instructional leadership, and (e) 
authentic, problem-based learning 
experiences (Darling-Hammond et al., 
2007; Jackson & Kelley, 2002) 

The importance of aligning the 
curriculum with professional standards 
is emphasized in much of the literature 
on school leadership education 
(Cambron-McCabe & McCarthy, 2005; 
Darling-Hammond et al., 2007; Hart & 
Pounder, 1999; Jackson & Kelley, 2002; 
Murphy, Elliott, Goldring, & Porter, 


2007; Young et al., 2002). The Interstate 
School Leader Licensure Consortium 
(ISLLC) standards are often touted as 
being effective and influential in 
guiding the design of 'exemplary' 
programs. According to Darling- 
Hammond et al. (2007), " [a]t least 46 
states... have adopted the ISLLC 
standards for principal preparation as 
part of their program approval process" 
One important and defining feature of 
the ISLLC model noted in the literature 
is that the standards focus on the 
importance instructional leadership and 
school improvement. 

Emphasizing instruction as a part 
of leadership necessitates that principals 
see themselves as educators rather than 
managers (Murphy, 2002). Instructional 
leadership encompasses "a school-wide 
focus on better teaching and learning" 
(Wallace Foundation, 2008, p. 2). 
Student learning success is highlighted 
and the school leader, along with 
teachers, develops a school vision that 
incorporates learning success as a 
school-wide goal. 

Given the importance of 
excellence in instructional leadership, 
educational programs need to ensure 
they are matriculating quality 
candidates. Thus, a rigorous candidate 
selection process (Young et al., 2002; 
Murphy, 1999) is important. Darling- 
Hammond et al. recommend that 
leadership education programs actively 
recruit teachers who are proficient in 
teaching, exhibit leadership ability, and 
who represent the community in which 
they reside and teach. Active 
recruitment of high quality candidates is 
vital to educational program success as 
"the knowledge and skills of those who 
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enter a program determine to a great 
extent what kind of curriculum can be 
effective and what kind of leader will 
emerge" (p. 21). 

Successful educational leadership 
programs provide social and 
professional support through a cohort 
structure that facilitates collaborative 
research, group learning, reflective 
practice, and sense of community 
(Burnett & Muth, 2008; Darling- 
Hammond et al., 2007; Kelly & Jackson, 
2002). Barnett and Muth contend it is a 
moral obligation to facilitate program 
completion. The cohort structure yields 
not only higher completion rates, but 
also helps students construct 
professional networks (Jackson & 
Kelley) and creates scholar- 
practitioners, in part, through action 
research experiences. 

Action research and field-based 
experiences are avenues through which 
students are exposed to authentic 
problem-based experiences. Many 
educational leadership scholars 
emphasize the importance of these types 
of learning experiences (Goodwin, 
Cunningham, & Eagle, 2005; Leithwood, 
Louis, Anderson, & Wahlstrom, 2004; 
McCarthy, 1999; Murphy 1999; Hart & 
Pounder, 1999; Van Meter, 1999), 
learning methods considered to be 
"powerful pedagogical practices, rooted 
in adult learning" (Orr, 2006, p. 495) 
Other such powerful practices include 
experience-based learning, reflective 
practice, and structures through which 
communication and collaboration are 
facilitated (Orr, 2006; Barnette & Muth, 
2008; Goodwin et al., 2005; Kottkamp & 
Silverberg, 2003; Murphy, 2006) 


In the next section the evaluation 
plan for the Auburn University 
Educational Leadership Master's 
Program is reviewed, linking program 
elements to evaluation approaches. 
These components and approaches are 
discussed using overarching framework 
with three major components: 
collaboration, comprehensiveness, and 
the on-going process of evaluation as a 
driving force for program reform. 

The Educational Leadership Program 
Redesign and Integrated Evaluation 

Patton (1996) argues that 
evaluation should be broadly 
conceptualized as a knowledge¬ 
generating activity. In Patton's view, 
evaluation is "too rich and varied to be 
pigeon-holed into a single dichotomy" 
(A Menu of Evaluation Purpose section, 
paragraph 2). Along these lines, the 
Auburn University Educational 
Leadership Team and its partners 
conceptualized evaluation for Auburn 
University's Educational Leadership 
Master's Program to be an activity that 
incorporates a variety of approaches 
and methods to accommodate different 
evaluation needs. Hence, the leadership 
team and partners adopt facets of 
program evaluation models that 
facilitate collaboration, emphasize 
learning, and generate knowledge that 
can be used to inform program 
implementation processes, such as 
quality of internship experience; 
progress, for example, mid program 
review of student progress, and 
products, such as quality of K-12 school 
leadership of Auburn University 
graduates (Guskey, 2001; Reed, Kochan, 
Ross, & Kunkel, 2001) such that the 
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program evolves to meet on-going 
needs. 

The redesigned Educational 
Leadership Master's Degree Program 
incorporates the features of exemplary 
schools described in the literature 
reviewed previously with revised 
admission procedures that emphasize 
candidate quality, curriculum that is 
developmental and field-based, and a 
cohort structure that facilitates 
collaboration (e.g. Jackson & Kelley, 
2002). The evaluation plan for the 
program borrows from (a) objectives- 
oriented, (b) management-oriented, (c) 
consumer-oriented, (d) expertise- 
oriented, and (e) participant-oriented 
approaches evaluation approaches. In 
the succeeding section, the Auburn 
University Educational Leadership 
Program evaluation plan elements in 
conjunction with the approach utilized 
is discussed with emphasis on the 
collaborative, comprehensive, and on¬ 
going aspects of the plan. 

Collaboration. A strong 
proponent of the consumer-oriented 
approach is Michael Scriven (1991,1994, 
1996, 1997, 1998, 1999; Scriven & Coryn, 
2008). He is concerned with the impact 
the evaluation has on program 
participants and other stakeholders 
stating that 

it [consumer-oriented evaluation] 
regards the main function of an 
applied evaluation field to be the 
determination of the merit and 
worth of programs (etc.) in terms 
of how effectively and efficiently 
they are serving those they 
impact, particularly those 
receiving —or who should be 


receiving — the services the 
programs provide, and those 
who pay for the program (1994, p 
161). 

Tineke and Stake (2001) emphasize that 
different stakeholders have different 
expectations and values and that these 
should be considered by evaluators. 
One approach to integrating stakeholder 
voices is participatory evaluation. 
Cousins and Whitmore (1998) state that 
"[participatory evaluation implies that, 
when doing an evaluation, researchers, 
facilitators, or professional evaluators 
collaborate in some way with 
individuals, groups, or communities 
who have a decided stake in the 
program, development project, or other 
entity being evaluated" (p. 5), and 
Young et al. (2002) assert that 
" [c] ollaboration is... a necessary 
ingredient for the improvement of 
preparation programs (p. 142)." These 
authors contend that collaboration with 
key stakeholders is fundamental to 
making substantive program changes 
that support the development of 
successful school leaders. 

The Auburn University 

Educational Leadership Team places 
heavy emphasis on participatory 

approaches as it considers collaboration 
with all stakeholders to enhance team 
members' ability to assess, judge, and 
reform the program as needed. 
Collaboration with partners began at the 
design stage: the team initiated the 
curriculum redesign efforts through 
conversations with partner school 
practitioners and students seeking to 
discover beliefs and assumptions of 
both Auburn University Educational 


487 



Ross / PROGRAM EVALUATION AND REFORM 


Leadership faculty and K-12 
stakeholders (Cambron-McCabe, 2003; 
Young et al., 2002). The conversations 
led the leadership team and partners to 
develop admissions and curriculum 
changes closely tied to school and pK-20 
student learning needs (e.g. 
developmentally appropriate 

instruction, practical application of 
material learned) and to state standards. 

Partners as stakeholders in the 
redesign. Planning for the redesign 
began in the fall of 2005 and in 
December of that year, Auburn 
University and its K-12 partners 
presented an overview of the plan to the 
governor's Congress Implementation 
Committee. In early 2006, four 
committees were established to address 
curriculum, admissions, partnerships 
and advisory council, and evaluation 
and the committees met throughout the 
summer and fall of 2006 and the spring 
and summer of 2007. Through these 
committees, the Auburn University 
Educational Leadership Team, K-12 
partners, and leaders from the ALSDE 
developed a vision for the program and 
established an innovative new 
curriculum. The vision emphasized the 
development of instructional leaders 
(Murphy, 2002; Wallace Foundation, 
2008) — leaders that have the capacity to 
improve education for all and to lead 
schools with a strong focus on student 
learning. The curriculum was based on 
a thorough review of the literature 
pertaining to leadership program 
design, leadership effectiveness, and 
ALSDE Standards for Instructional 
Leaders. The Auburn University team 
and partners also developed an 
evaluation plan that incorporated on¬ 


going student, instructor, and course 
assessments. Residency internship 
requirements were outlined along with 
internship assessment guidelines. 
Additionally, the team and its partners 
developed guidelines for assessment 
portfolios that students build through 
coursework, wrote an internship 
handbook, and agreed that assessment 
would include a capstone event in 
which students would present action 
research (completed through 

coursework) to program faculty, school- 
based partners, and fellow cohort 
students and receive feedback. Our first 
cohort of students began the program in 
the summer of 2008, with a completion 
date of the summer of 2009. 

Students as stakeholders in 
evaluation. Auburn University 
Educational Leadership faculty assumes 
that the Auburn Leadership graduate 
students, as future school leaders, 
genuinely want to improve the quality 
of public school education for their K-12 
students and understand what they 
need in terms of professional growth to 
do so. Therefore, faculty and 
administrators must listen to students' 
in order to fully understand what they 
are facing in their public school settings. 
They must also ascertain what the 
graduate faculty does or does not do for 
them in terms of learning to lead 

effective schools. Thereby, faculty and 
administrators empower them to help 
shape the direction of the program. For 
example, the Educational Leadership 
team seeks their feedback at the 

formative stage through course 

evaluations. In addition, students are 
also given voice through seminar 

discussions and a learning community 
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consisting of university faculty, 
students, and K-12 administrators from 
partner school districts. 

Partner school districts as 
collaborators. The team, additionally, 
draws on the expertise of partners as 
public school consumers of the 
program's product. Input and insights 
of field-based coaches who work with 
internship students, as well as students, 
the consumers, help the faculty evaluate 
individual courses, field internships, the 
graduate participants, faculty, and 
program implementation. 

The leadership team considers 
collaboration with partners essential at 
the student application stage. This is 
consistent with the recommendations of 
such researchers as Darling-Hammond 
et al. (2007) and Young et al. (2002) who 
underscore the need to recruit quality 
students (e.g. excellent teachers with 
strong communication skills) who are 
team players and who have 
demonstrated leadership skill (e.g. 
ability to take the lead on school 
committees). Toward this end, the 
interview process includes student 
demonstration of individual and group- 
based problem solving as well as 
presentations, which are collaboratively 
evaluated based on the expertise of both 
university and K-12 administrators. The 
faculty continues to draw on the 
expertise of the partners through the use 
of field-based coaches and school 
administrators who provide input about 
student internship experiences and 
follow-up assessment which includes 
employer surveys which become a part 
of an overall comprehensive program 
review that addresses admissions. 


students, instructors, and the 
curriculum. 

Comprehensive Evaluation 

Design. A comprehensive evaluation 
plan must include all program processes 
and outcomes. In conceptualizing our 
evaluation, the leadership team and its 
partners borrow from Stufflebeam's 
management-oriented CIPP approach 
(Stufflebeam & Shinkfield, 2007) which 
encompasses context (C), input (I), 
process (P), and product (P) decisions, 
and spans the planning stage through 
program completion and subsequent 
leadership effectiveness of our 
graduates. CIPP also includes 
assessments of students, instructors, and 
the curriculum. Context evaluation 
encompasses identifying the target 
population and the needs or problems 
to be addressed. Much of this work was 
completed in the planning stage. 

Context evaluation. Context 
evaluation begins with the planning of 
the redesign. Thus, at the program and 
evaluation design (planning) stage 
(Guskey, 1999) K-12 partners were 
sought through the creation of an 
Advisory Council (university faculty, K- 
12 practitioners, an ALSDE 
representative, and students) and an 
Admissions Committee (Auburn 
University faculty and K-12 
administrators). Through collaborating 
with these partners, school hiring and 
leadership needs were identified. 
Consistent with the literature 
summarized in this paper, leadership 
needs centered on the development of 
sound admissions criteria, alignment of 
curriculum with standards, and school 
improvement through student-centered 
instruction (Barnett & Muth, 2008; 
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Cambron-McCabe & McCarthy, 2005; 
Darling-Hammond et al., 2007; Hart & 
Pounder, 1999; Jackson & Kelley, 2002; 
Murphy, 2002; Orr, 2006; Wallace 
Foundation, 2008; Young et al., 2002). 
Following this, the input evaluation step 
involves evaluating and redesigning the 
existing program based on identified 
needs, including need for quality school 
leaders and the need for student- 
centered instruction. An initial step is to 
look at the literature on leadership 
education and the standards for quality 
in the programs. 

Input evaluation. A part of 
redesigning the Auburn University 
educational leadership program was to 
align our curriculum with state 
standards as well as ISLLC and the 
NCATE standards. The purpose of 
accreditation boards such as NCATE is 
to provide oversight using an expertise- 
approach to evaluation with site visits 
by 'expert' teams. Accreditation helps 
ensure that the educational leadership 
program and curriculum maintain high 
standards that provide a foundation for 
high quality educational leadership 
graduates, the purpose for the redesign. 
Thus, the leadership team redesigned 
the curriculum, incorporating ALSDE 
state standards into course goals and 
objectives. These standards address: 

(1) Planning for continuous 
improvement. Evaluation must 
be continuous so that the 
Auburn University Educational 
Leadership Team and partners 
can refine and improve the 
program as issues are 
identified. 


(2) Teaching and learning 

practices. The Auburn 

University Leadership faculty 
emphasizes instructional 

leadership that maintains 

student learning must be the 
focal point for all educators. 

(3) Human resources development. 

For example. Auburn 

University leadership faculty 
members seek to develop 
educational leadership 

graduates that demonstrate 
ability to recruit, select, 
evaluate, and mentor new 
faculty in their schools. 

(4) Diversity. The team strives to 
ensure that all of educational 
leadership graduates have the 
capacity to respond to a variety 
of contexts, such as political, 
social, cultural, and economic 
contexts. 

(5) Community and stakeholder 

relations. Graduates of the 
Auburn University Educational 
Leadership program are 
expected to see schools as a part 
of a larger community which 
necessitates collaboration 

among all groups of 
stakeholders (e.g. parents, 
community agencies). 

(6) Technology. Educational 

leaders that graduate from the 
Auburn University graduate 
program will demonstrate 
ability to integrate technology 
as tools for managing, teaching, 
and communication. 

(7) Management of the learning 
organization. To be good 
managers. Auburn University 
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leadership faculty expects our 
graduates to have the ability to 
manage components of the 
organization including 

facilities, financial resources, 
and the learning environment. 

(8) Ethics. Faculty expects that its 
graduates guide their school 
policies and practices with 
honesty, integrity, and fairness. 
(Governor's Congress on School 
Leadership Final Report, 2005) 

These standards, as well as 
standards set forth other educational 
leadership organizations, such as 
NCATE, ISLLC, Southern Regional 
Educational Board (SREB), and the 
Educational Leadership Constituent 
Council (ELCC), are identified as goals 
to be aspired to in our leadership 
program; thus, the Auburn University 
Educational Leadership Team draws on 
the objectives-oriented approaches 
developed by Tyler (1991) to assess the 
alignment between the ALSDE 
leadership preparation standards and 
the curriculum. Advocates of this 
approach base the evaluation on goals 
and objectives and collect data that 
inform program decision makers about 
the extent to which these goals and 
objectives were met. Sanders and 
Cunningham (in Fitzpatrick et al., 2004) 
specifically recommended both logical 
and empirical methods, including 
content analysis of records such as 
meeting minutes or syllabi. To evaluate 
the alignment between curriculum and 
state standards, the Auburn University 
faculty, along with its partners, created 
a "crosscheck" matrix to align the state 
standards (Appendix A) with courses. 


Additionally, the team reviewed syllabi 
as they were developed to assess 
whether courses addressed all 
standards. To further check adherence 
to these standards, the evaluation plan 
calls for the collection and analysis of 
student work products (portfolio) and 
the solicitation feedback from K-12 
partners after student engagement in 
field-based activities. For example, 
surveys disseminated to both students 
and K-12 partners and students can 
contain items that directly ask about 
how activities addressed specific 
standards as well as perceptions of 
student progress. 

Process evaluation. Process 
evaluation takes place continuously and 
encompasses the evaluation of students, 
instructors, courses, and program 
structures. Student evaluation begins 
with a rigorous comprehensive 
admissions process: candidates submit 
an application portfolio that includes 
evidence of accomplishments as an 
education as well as a letter of intent 
and recommendations. Candidates also 
interview with a team of faculty 
members and administrators from 
partner schools as a part of the process. 
At the interview, candidates work in 
teams to problem-solve and then 
present their solutions to faculty and 
school partner representatives. In 
addition, an on-site writing example is 
required as part of the admissions 
process. Through a rigorous admissions 
process, the level at which candidates 
demonstrate strong teaching and 
leadership ability, willingness to work 
hard and collaborate with others, and 
communication proficiency is assessed. 
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Once admitted, students attend a 
three-day summer institute to orient 
themselves to the program. By bringing 
in well-known guest presenters and 
incorporating small group discussions 
and activities related to topics 
presented, the institute also acts as a 
professional development workshop the 
leadership program can offer to faculty 
and school-based partners such as field- 
based coaches, mentors, and Advisory 
Council members. Evaluations of the 
institute by participants help faculty and 
partners refine the design and plan for 
future summer institutes and also sets 
the stage for student development 
through an intense schedule of courses 
over three semesters. 

The team accentuates 
instructional leadership in the 
coursework... school leadership that 
places much value and emphasis on the 
teaching and learning process and 
authentic or "real world" learning 
experiences (Darling-Hammond et al., 
2007; Goodwin et al., 2005; Leithwood et 
al., 2004). Throughout coursework, 
student cohorts are tasked with 
authentic group think tank projects 
through which the processes of content 
knowledge and skill acquisition, 
application of learning, cognitive 
complexity, and professional 
dispositions, such as openness, positive 
attitude, and belief that all students can 
learn (Murphy et al., 2007), are assessed 
by faculty as content experts and judged 
by field-based leadership experts such 
as school administrators. 

In addition to instructor 
assessments of student acquisition of 
course content, students formally self 
assess as part of their residency 


internship. The internship consists of 10 
consecutive days in each summer and 
five consecutive days during the fall and 
the spring semesters. Students develop 
an internship plan for personal growth 
(Appendix B) and keep an activity log 
(Appendix C), collect artifacts, and pre- 
and post-rate their progress toward 
meeting standards and goals. See 
Appendix D for internship portfolio 
checklist. 

Students also participate in the 
evaluation process through assessing 
individual courses and instructors. 
Instructors can the use this feedback to 
refine course content and adjust 
teaching activities. Further, students 
and school-based partners, as 
stakeholders, assess satisfaction with 
progress and outcomes (products) 
through written feedback and surveys 
as well as on-the-job assessments. 

Product evaluation. Outcome 
evaluation is a test of overall program 
product quality. For example, at the end 
of their studies, students are required to 
demonstrate learning to faculty and K- 
12 practitioners from local partner 
districts through a capstone 
presentation of think tank project 
outcomes and recommendations to 
faculty, the district administrative 
teams, and cohort group members. 
Outcomes assessed include, along with 
dispositions such as a deep appreciation 
for diversity, knowledge and skill 
acquisition and impact on both school 
improvement and student achievement. 
End of program written student 
feedback, surveys, and follow-up 
satisfaction surveys after graduation 
provide summative outcome 
evaluations of the program that then can 
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be used to inform program reform and 
refinement for future students. 

Ongoing Evaluation. Scriven 
(2007, 1991) defines two types of 
evaluation, summative and formative. 
He states that "[f]ormative evaluation is 
evaluation designed, done, and 
intended to support the process of 
improvement," and that '[s] summative 
evaluation...is done for, or by, any 
observers or decisions makers...who 
need evaluative conclusions for any 
other reasons besides development" (p. 
20). Given that the evaluations of 
students, faculty, partnerships, and 
program components are all used to 
inform program or student 
development decisions, even those 
evaluations at concluding points, such 
as at the end of a semester, can be 
considered formative and in the service 
of on-going improvement. However, in 
the following sections, specific student 
and program on-going, formative 
assessment tools, and activities are 
highlighted. 

Student assessment. The student 
portfolio acts as both a formative and 
summative assessment tool. It is 
formative in that students receive 
feedback on progress demonstrated by 
the artifacts throughout the program, 
and it is summative in that it acts as a 
final assessment tool at the end of the 
program. The internship portfolios are 
developed over the course of four 
separate internship residencies. As each 
residency is completed, artifacts are 
added to the portfolio and progress 
toward meeting internship goals is 
assessed. Student self assessments of 
their progress includes a description of 
the activity and a reflection statement 


(Appendix C). Field-based coach 
assessments include ratings of 
performance, a description of strengths, 
and a statement of areas for continued 
growth (Appendix E) 

Another student learning and 
assessment activity that the faculty 
conducts on an on-going (semester) 
bases is the think tank. For example, 
during the 2008-2009 year, students 
researched problems or issues related to 
proration, school supports, and 
extracurricular activities. Results of the 
think tank research are presented to K- 
12 practitioners from local partner 
districts and faculty at the end of the 
semester. These on-going assessments, 
along with other assessments of 
students and faculty, provide data that 
informs the leadership team of the 
overall effectiveness of the educational 
leadership program. 

Program assessment. Program 
assessments are conducted each 
semester through Advisory Council 
meetings. These meetings bring together 
College of Education faculty and school- 
based personnel to discuss and assess 
what has worked and what has not 
worked so that the leadership team and 
its stakeholders can improve the 
program to address hiring needs of 
schools. Topics of discussion at the last 
Advisory Council meeting included 
think tank activities and internships. 
Practitioners and school leaders from 
partner districts reported on student 
internship activities and their 
assessments of the usefulness of the 
activities the student participated in 
terms of learning. These K-12 
practitioners and leaders were also 
asked about what future think tank 
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research they would consider useful to 
them, which triggered discussion about 
the need for data base where all 
education related data can be readily 
accessible. The practitioners and leaders 
suggested that think tank research 
might include investigating what 
educational data might be useful and 
how to organize it. Activities like the 
think tank provide avenues through 
positive educational reform or 
transformation can take place. 

Preskill and Torres (2000) 
advocated the adoption of 
transformative goals that facilitate 
learning. They state that 
'[transformative learning can be 
facilitated when employees seek to 
understand something, address critical 
organizational issues, and improve their 
work through a participatory, dialogic, 
reflective, and inquiry-oriented 
approach to evaluation and the use of 
findings." (p. 29). Reflection, inquiry, 
and dialogue are routinely a part of 
scheduled Educational Leadership 
Team meetings and the Advisory 
Committee meetings, through which 
self-assessment and self-evaluation is 
conducted to make program process 
reform (implementation) decisions on 
an on-going basis. 

Discussion 

Conclusions. As Patton (1996) 
advocates, the graduate program 
evaluation is designed to integrate rich 
and varied sources of information for 
the purpose of generating knowledge 
that can be used to inform, define, 
reform, and transform (Reed et al., 2001) 
the program, participants, faculty, and 
stakeholders. Though some researchers 


are purists and do not integrate 
qualitative and quantitative methods, 
many researchers and evaluators use 
some blend (Ross, Narayanan, Hendrix 
& Myneni, 2009). The Auburn 
University Educational Leadership 
Team took a pragmatic approach to 
designing the evaluation and drew from 
many evaluation approaches to identify 
the best methods based on questions 
asked, context, and stakeholder needs, 
and often an evaluation strategy was 
conceptualized through more than one 
evaluation lens to evaluate all program 
components such as individual courses 
and field internships, program 
participants and faculty, and program 
implementation. Additionally, the 
Auburn University faculty and partners 
will continue to utilize a wide variety of 
data collection methods, including 
questionnaires, observations, 

examinations, portfolios, reflections. As 
Sander (2002) recommends, in order to 
sustain a high-quality program the 
evaluation became a part of the 
program's culture. 

When evaluation is part of the 
culture of the program, it is on-going 
and intertwined with all the program 
components, stakeholders, and 
structures. The Auburn University 
Educational Leadership team began the 
redesign of the master's program 
through a review of literature and 
seeking input from local K-12 
practitioners. The knowledge gained 
from these activities formed the basis for 
defining a new program. Collaboration 
with both K-12 practitioners from local 
school districts and the ALSDE 
representatives was essential as the 
evaluative feedback they provided 
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throughout the planning process and 
initial implementation of the program 
helped shape the program such that the 
graduates are highly prepared to 
become leaders that make a difference 
in student learning. On-going 
evaluation is needed to continue to 
refine and transform the program to 
meet stakeholder emerging and 
changing needs. 

Initial Assessments. The Auburn 
University Educational Leadership 
Master's program began 

implementation in the summer of 2008. 
The first cohort is now in their second 
semester of coursework and all 16 of the 
students that matriculated as continuing 
in good standing. Residency portfolios 
have been assessed with all students 
meeting their goals and the expectations 
of the field-based coaches and faculty 
supervisor. Initial think tank research 
outcomes have been presented and well 
received, and the next presentation will 
take place at the summer institute 
orientation meeting where participants 
will formally rate all program 

components, including the student 
presentations. Evaluation of the 

program by K-12 practitioners has 
generally been positive and useful. Due 
to their feedback in the recent Advisory 
Council meeting information about 
graduates will be distributed to school 
districts for hiring purposes. Student 
evaluations of the program have also 
been generally positive and useful. 
Based on feedback the graduate 

students, course sequencing changes 
will take place to better prepare 
students in a timely manner for their 
licensure exam in the spring. 


Recommendations. Any 
graduate level educational leadership 
faculty seeking to continually reform 
and transform to better meet the needs 
of their stakeholders should engage in 
discussions about their underlying 
assumptions and needs in order to 
develop a coherent and consistent 
program. The assumption that because 
education in multi-faceted and changing 
endeavor leading to the need to 
confront and challenge the status quo 
was an assumption this author made 
that wasn't necessarily shared by all 
others. Throughout the course of the 
initial implementation the team has 
always been able to discuss issues and 
concerns and come to a consensus about 
how to proceed. Other assumptions do 
seem to be held by most or all team 
members. The faculty and its partners 
did confront existing paradigms and 
methods by completely redesigning the 
program from scratch and incorporating 
innovative methods such as 
coordinating projects across courses and 
semesters, building in multiple 
opportunities for authentic field-based 
learning activities, and emphasizing 
instructional and curricular leadership 
rather than a management style 
leadership. Because of the willingness to 
challenge existing paradigms, the 
leadership team and its partners built in 
multiple approaches to evaluation to 
accommodate different data uses and 
evaluation purposes. 

In the field of education, 
collaboration appears to be a 
particularly effective approach to 
designing and evaluating programs. At 
the higher education level, collaboration 
often takes place across universities by 
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individual or groups of researchers who 
seek to strengthen conclusions. 
Collaboration at the K-12 level 
strengthens programs through the 
sharing of ideas and resources. For 
example the Auburn University Truman 
Pierce Institute directed by Cindy Reed 
creates opportunities for students, 
teachers, and school leaders to meet and 
discuss problems, issues, and solutions. 
Given that evaluative decisions in 
education are often made for program 
improvement purposes rather than 
decisions related to program 
continuation, this author recommends 
that educational program evaluation 
efforts include not only program 
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Appendix A 

Course 'crosscheck' with State Standards 

This matrix cross-references each of the Alabama Standards with program courses and curricula. 


Alabama 


Summer 1 


Fall 


Spring 


Summer II 


Standard 
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Action 
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Lead and motivate staff, students, and families to achieve the school's vision 
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Work with faculty to identify instructional and curricular needs that align with vision and resources 
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(2Ha)2.(w) 

interact with the community concerning the school's vision, mission, and pnonties. 
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Work with staff and others to establish and accomplish goals 
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Relate the vision, mission, and goals to the instructional needs of students 
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Use goals to manage activities 
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Use a variety of problem-solving techniques and decision-making skills to resolve problems 
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Focus upon student learning as a driving force for curriculum, instruction, and institutional decision-making 
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Use a process for gathenng information to use when making decisions 
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Use multiple sources of data to manage the accountability process 
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Assess student progress using a variety of techniques and information 
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Monitor and assess instructional programs activities, and matenats 
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Use external resources as sources for ideas for improving student achievement 
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Collaborate with commun*y. staff, district, state, and university personnel to develop the instructional program 
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Align curriculum, instructional practices, and assessments to district, state, and national standards 
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Appendix B 

Auburn University Instructional Leadership Program 
Spring Semester 2009 Internship Plan 

This plan specifies the standards/indicators that will guide the internship for this 
semester. Under each standard/indicator are listed activities that the intern should 
engage in during the internship. It is essential that interns are allowed ample 
opportunity to observe, participate and lead activities from this list. 

Standard: Develop a written plan of action for the development of a vision for 
comprehensive integration of technology into a school curriculum 
Suggested Activity: Participate in a school-level committee discussion of technology 
needs and develop/submit a plan of action for integrating technology in the coming 
school year based on school needs/faculty input 

Standard: Apply research in the area of management 

Suggested Activity: Lead an analysis of building level "non-instructional" services and 
costs at the direction of the principal and develop a profile of expenditures in 
management areas that can be evaluated for cost savings. 

Standard: Participate in designing a class schedule utilizing appropriate technological 
resources 

Suggested Activity: Observe and/or participate in a review of class schedule design for 
2009-2010 school year based on budget forecasts, program priorities and existing 
staffing patterns (focusing on such areas as earned teacher units versus needs based on 
programming decisions). 

Standard: Knowledge of political factors which impact on schools and school systems 
Suggested Activity: Interview (may be done as a small group) a central office 
administrator and a school level administrator regarding funding processes at the state 
and local levels, including political realities of state funding and how decisions made at 
those levels impact the decisions administrators must make regarding staffing, program 
offerings, etc. 
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Appendix C 

Auburn University 
College of Education 
EDLD 7930 

Administrative Intern Activity Log 

Name: Semester: 

Date of Activity: 

School Site: 

*Type of Experience: Observing Participating Leading 
Activity #:_ 


Activity Title: 


Standard #/Key Indicator 

COURSE 

# 












Type Standard # only. Write out the key indicator (Alabama Instructional Standards handout). 
Course # for Summer will either be EDLD 7540 Curriculum or EDLD 7530 Continuous 
Improvement. If you engaged in an activity that doesn’t fit either course, leave it blank. 
Description of Activity: 

Reflection: 

Action Taken or Action Needed: 

Artifacts for the activity should follow this Activity Log fonn in the portfolio. 

*There must be no more than: 

50% observation 

and at least: 30% participating & 

20% leading 
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Appendix D 

Administrative Internship Portfolio Checklist 

_Competency Verification Form-pages 1 & 2 (One fonn for the year.) 

_Cooperating Administrator’s Evaluation Form w/signatures- page 3(each internship 

assignment) 

_Professional Growth Statement (1 for Summer and Fall, 1 for Spring and Summer 2009) 

_Activity Log Form (as many as you need) 

_Artifacts to support Activity Log Form (schedules, programs, letters, data sheets, power 

points...) 

_Internship Plan (with signatures) 

_EDLD 7930 Pre- Post- (Should only have pre- rated now. Will rate post- after your last 

internship.) 
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Appendix E 

Field-based Coach Evaluation Form 

Auburn University 
College of Education 
EDLD 7930 - Residency Internship 
Field-based Coach Evaluation Form 

The graduate student is to include this form in his/her portfolio and submit it to the Clinical Professor at Auburn 
University at the end of the internship. 

Clinical Professor: Dr. Lynne Patrick 

Clinical/Associate Professor 
EFLT Department 
Auburn University, AL 36849-5221 
rlp0004@auburn. edu 

334-844-3024 

Brief description of what the intern has done. 


What strengths have you observed in working with the administrative intern? 


What areas of continued growth would you suggest? 

What is your overall rating of the administrative intern’s performance this semester? Circle one: 
Excellent Satisfactory Unsatisfactory 

Please explain your rating: 


Field-based Coach’s Signature:_Date: 

Administrative Intern’s Signature:_Date: 
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